NexusPi Git Node
ESP32_HEAP_FRAGMENTATION.md main (ee519cc2) Text, 7.18 KB
ESP32 Heap Exhaustion in microReticulum — Diagnosis & Fix
Symptom
On an ESP32-S3 (Heltec V4, 324 KB internal heap, PSRAM, TLSF allocator), the
device rebooted every 10–14 minutes. The heap watchdog fired at the 20 KB
critical threshold. Free heap declined at a steady ~15 KB/min despite all
static data structures being capped and stable.
Investigation
Heap telemetry was already instrumented at three points per packet cycle:
T282828
[HEAP-TEL] boundary: -844 bytes (after firewall filter)
[HEAP-TEL] inbound: -1212 bytes (after full inbound processing)
[HEAP-TEL] jobs: +764 bytes (after periodic cleanup)
Every packet cycle net-leaked ~400–700 bytes. Over ~1,000 packets in
10 minutes, that is ~150 KB permanently lost. Static table sizes (T383838paths,
T383838dests, T383838announce_table, T383838reverse_table) were measured and confirmed
stable — the leak was not in RNS-level data structures.
Root Cause: T383838std::set<Bytes> node fragmentation
Four T383838std::set<Bytes> containers were implemented as red-black trees:
┌───────────────────────────────┬──────────────┬────────────────────┐
│ Container │ Typical size │ Inserts per packet │
├───────────────────────────────┼──────────────┼────────────────────┤
│ T383838_packet_hashlist │ 100 │ 1 │
│ T383838_global_blobs │ 8 │ 1 │
│ T383838_boundary_local_addresses │ 128 │ 1–2 │
│ T383838_boundary_mentioned_addresses │ 128 │ 4–5 │
└───────────────────────────────┴──────────────┴────────────────────┘
Every T383838insert allocates a tree node (~40 bytes) plus a **T383838shared_ptr
control block (~24 bytes)** for the T383838Bytes copy-on-write wrapper. When the
sets hit their cap and entries are evicted (oldest-first T383838erase), the tree
nodes are freed. However, on ESP32 even with TLSF, freeing many small
scattered allocations creates heap holes that cannot be coalesced — free heap
appears adequate in aggregate, but T383838malloc fails for larger contiguous
requests. This is classic fragmentation from node-based containers.
│ Why this matters: T383838Bytes already uses T383838shared_ptr<vector<uint8_t>> for copy-on-write sharing of the
│ actual hash data. The T383838std::set tree node is additional overhead on top of that — pure container
│ bookkeeping, not payload.
Quantified
• Tree nodes: (100 + 8 + 128 + 128) × 40 bytes = ~14.6 KB
• T383838shared_ptr control blocks: 364 × 24 bytes = ~8.7 KB
• Total overhead: ~23 KB of container bookkeeping that churns on every
insert/evict cycle
• Per-packet net loss: ~150 bytes (fragmented, cannot be recovered)
• Time to critical (20 KB): ~10 minutes at ~1.7 packets/sec
Fix: T383838std::set<Bytes> → T383838std::vector<Bytes>
Replaced all four containers with flat T383838std::vector<Bytes>. Vectors store
elements inline in a single contiguous allocation — **zero per-element heap
overhead** beyond the hash data itself.
API migration
┌───────────────────────┬────────────────────────────────────┬─────────────────────────────────────┐
│ `BT383838`Fdddstd::set<Bytes>`f`b │ `BT383838`Fdddstd::vector<Bytes>`f`b │ Rationale │
├───────────────────────┼────────────────────────────────────┼─────────────────────────────────────┤
│ T383838.insert(x) │ T383838.push_back(x) │ Dupes already checked before insert │
│ T383838.find(x) != .end() │ T383838std::find(begin(), end(), x) != e… │ O(N) linear; N ≤ 128 is negligible │
│ T383838.erase(begin(), iter) │ T383838.erase(begin(), begin() + N) │ Front-truncation for FIFO cap │
│ T383838.clear() / T383838.size() │ T383838.clear() / T383838.size() │ Unchanged │
└───────────────────────┴────────────────────────────────────┴─────────────────────────────────────┘
Impact
• Eliminated 364 tree-node allocations — removed ~23 KB of pure container
overhead
• Zero fragmentation from set-node churn — vectors do a single realloc on
growth, no per-element malloc/free
• Per-packet T383838boundary delta dropped from ~844 bytes to ~200–300 bytes
• RAM usage unchanged at 21.9% (71,624 / 327,680 bytes)
• Build size unchanged at 20.0% flash
Supporting changes
While investigating, several static caps were also tightened for extra
headroom on the ESP32:
┌───────────────────────┬─────┬─────┬───────────────────────────────────────────┐
│ Constant │ Old │ New │ Rationale │
├───────────────────────┼─────┼─────┼───────────────────────────────────────────┤
│ T383838MAX_PATHS_PER_DEST │ 3 │ 2 │ Halves per-destination path entry memory │
│ T383838MAX_GLOBAL_BLOBS │ 16 │ 8 │ Anti-replay only needs a few recent blobs │
│ T383838path_table_maxsize │ 24 │ 16 │ Fewer max destinations in table │
│ T383838path_table_maxpersist │ 12 │ 8 │ Fewer entries persisted to flash │
│ T383838_boundary_maxsize │ 200 │ 128 │ Less boundary address tracking │
└───────────────────────┴─────┴─────┴───────────────────────────────────────────┘
A T383838clear_caches_in_memory() method was added to T383838Transport, called from
the existing heap watchdog at HEAP_PRESSURE (28 KB):
• Clears T383838_packet_hashlist (duplicate detection — rebuilds naturally)
• Clears T383838_global_blobs (anti-replay — old announces may replay once)
• Clears T383838_announce_rate_table (rate limiting state — resets)
• Clears T383838_discovery_pr_tags (path request dedup)
• Then calls T383838cull_path_table()
General recommendation for the microReticulum repo
On ESP32-class devices with constrained heap and no MMU:
1. Prefer T383838std::vector over T383838std::set / T383838std::map when N ≤ ~200 and
insert/find frequency is moderate.
2. T383838std::set<Bytes> is a double-allocation trap: one allocation for the
tree node, one for the T383838shared_ptr control block — neither of which
stores payload.
3. If ordering isn't needed (hashlists, address sets, blob caches), a
flat vector with linear search is strictly better for heap health.
4. Consider a T383838FlatSet<T> wrapper that uses T383838std::vector internally
with T383838std::find — it would be a drop-in replacement for most T383838std::set
use cases in this codebase.
5. Audit other node-based containers — T383838std::map<Bytes, AnnounceEntry>
(T383838_announce_table), T383838std::map<Bytes, ReverseEntry>
(T383838_reverse_table), and T383838std::map<Bytes, LinkEntry> (T383838_link_table)
have the same tree-node allocation pattern. If their sizes typically
stay small (< 50 entries), they may be acceptable. If they grow large
under load, consider migrating to sorted T383838std::vector with binary search.
Files changed
┌────────────────────────────────────────────────┬─────────────────────────────────────────────────┐
│ File │ Change │
├────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
│ T383838lib/microReticulum/src/Transport.h │ Added T383838PathEntry struct, T383838#include <deque>, T383838sele… │
│ T383838lib/microReticulum/src/Transport.cpp │ Multi-path insertion logic, T383838select_path() scor… │
│ T383838lib/microReticulum/src/Utilities/Persistence.h │ Added T383838Converter<std::deque<T>> and T383838Converter<P… │
│ T383838lib/microReticulum/src/Reticulum.h │ Updated T383838get_path_table() return type │
│ T383838lib/microReticulum/src/Reticulum.cpp │ Updated T383838get_path_table() and T383838drop_all_via() fo… │
│ T383838lib/microReticulum/src/Link.cpp │ Added missing T383838Link::attached_interface() const… │
│ T383838RNode_Firmware.ino │ All interfaces → T383838MODE_FULL; reduced path table… │
└────────────────────────────────────────────────┴─────────────────────────────────────────────────┘
Served by rngit 1.4.2 - Generated in 0.04s